Our app aims to help those with dietary restrictions manage their dietary restrictions when they encounter unfamiliar food. Sometimes, identifying ingredients in food can be difficult through normal methods such as checking labels or asking nearby staff due to factors like language barriers or insufficient signage. 

Unlike other apps that require outside information such as barcodes or crowd-sourced comments, our app aims to solve this problem through image classification that doesn’t require anything but a picture of the food itself to help our users with their dietary restrictions.

Once the user creates a set of dietary restrictions for our app to look out for, all the user has to do is take a picture of their food, and our app will use computer vision to determine whether that food is suitable to eat. 

Creating this set of dietary restrictions can be done with presets for more common restrictions such as maintaining a vegetarian diet, or specific ingredients can be selected to create a more specific profile of dietary restrictions. From there, the app is able to identify which ingredients are present in your dish and which, if any, of your dietary restrictions are in conflict with these ingredients.

After the initial brainstorming and prototyping phases, we were ready to get some input on our first designs. Conducting user interviews and holding a heuristic evaluation gave us tons of constructive feedback that we used to help improve our design.

The user interviews were incredibly useful in determining the challenges our target user group faced and which tasks they were having trouble with. For instance, one of our interviewees enjoyed making Chinese food but couldn’t read Mandarin, making shopping at Asian markets difficult, particularly when she was unable to communicate with the market’s employees due to language barriers. Another interviewee was concerned that the dining hall they ate at frequently had insufficient signage, and sometimes, the employees were unsure of which ingredients were present in the food. These observations, along with our analysis of similar apps, led us to determine that our app should be able to strive to perform image classification that doesn’t require anything but a picture of the food itself. Knowing this would be a difficult task, we added functionality for our app to let the user know if they needed to rearrange their food to get a picture where the ingredients were more distinct.

After receiving feedback from a group of evaluators during heuristic evaluation of a prototype we improved our adherence to the Android design standards and enhanced user flexibility and control. In particular, for user flexibility and control, we added feedback to notify users their actions were successful and options for recovering from error when creating a set of dietary restrictions. We also added a tutorial for first-time users to demonstrate the functionalities of our app.

From left to right, the drop down menu, once the general tab is selected, it will expand to show general dietary restrictions, and once “Vegan” is selected, in the box titled Current Dietary Restrictions, Vegan shows up.

Users are able to search for specific ingredients that our app should look out for. Once selected, the user can see their new dietary restriction in the “Current Dietary Restrictions” section as well as a check mark to confirm their selection was successful.

When the users want to remove a restriction they have selected, they can click the edit button on top right of “General Dietary restriction” and then click the “X” beside the restriction they want to deselect. The app will always automatically save the changes the users have made.

When the ingredients are not clearly visible in the dish, a notice will pop up telling the user to rearrange the food. After the user takes another picture of the food, it can then start to analyze.